ggml-vocab-llama-bpe.gguf.inp
1 ied 4 ½ months 2 __ggml_vocab_test__ 3 Führer 4 __ggml_vocab_test__ 5 6 __ggml_vocab_test__ 7 8 __ggml_vocab_test__ 9 10 __ggml_vocab_test__ 11 12 __ggml_vocab_test__ 13 14 __ggml_vocab_test__ 15 16 17 __ggml_vocab_test__ 18 19 20 21 __ggml_vocab_test__ 22 23 24 25 26 __ggml_vocab_test__ 27 28 29 __ggml_vocab_test__ 30 Hello world 31 __ggml_vocab_test__ 32 Hello world 33 __ggml_vocab_test__ 34 Hello World 35 __ggml_vocab_test__ 36 Hello World 37 __ggml_vocab_test__ 38 Hello World! 39 __ggml_vocab_test__ 40 Hello, world! 41 __ggml_vocab_test__ 42 Hello, world! 43 __ggml_vocab_test__ 44 this is 🦙.cpp 45 __ggml_vocab_test__ 46 w048 7tuijk dsdfhu 47 __ggml_vocab_test__ 48 нещо на Български 49 __ggml_vocab_test__ 50 កាន់តែពិសេសអាចខលចេញ 51 __ggml_vocab_test__ 52 🚀 (normal) 😶🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token) 53 __ggml_vocab_test__ 54 Hello 55 __ggml_vocab_test__ 56 Hello 57 __ggml_vocab_test__ 58 Hello 59 __ggml_vocab_test__ 60 Hello 61 __ggml_vocab_test__ 62 Hello 63 __ggml_vocab_test__ 64 Hello 65 Hello 66 __ggml_vocab_test__ 67 ( 68 __ggml_vocab_test__ 69 70 = 71 __ggml_vocab_test__ 72 ' era 73 __ggml_vocab_test__ 74 Hello, y'all! How are you 😁 ?我想在apple工作1314151天～ 75 __ggml_vocab_test__ 76 3 77 __ggml_vocab_test__ 78 33 79 __ggml_vocab_test__ 80 333 81 __ggml_vocab_test__ 82 3333 83 __ggml_vocab_test__ 84 33333 85 __ggml_vocab_test__ 86 333333 87 __ggml_vocab_test__ 88 3333333 89 __ggml_vocab_test__ 90 33333333 91 __ggml_vocab_test__ 92 333333333 93 __ggml_vocab_test__ 94 95 96 97 98 99 100 101 102 103 104 105 🚀 (normal) 😶🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български ''''''```````""""......!!!!!!?????? I've been 'told he's there, 'RE you sure? 'M not sure I'll make it, 'D you like some tea? We'Ve a'lL 106 __ggml_vocab_test__ 107 Việt 108 __ggml_vocab_test__