Spaces:

bigscience-data
/

bigscience-tokenizer

Runtime error

christopher commited on May 23, 2022

Commit

2954f61

1 Parent(s): 3920380

Update app.py

Files changed (1) hide show

app.py CHANGED Viewed

@@ -12,6 +12,6 @@ button_clicked = st.button("Tokenize")
 if text or button_clicked:
     st.write(len(tokenizer.tokenize(text)))
     st.components.v1.html(visualizer(text), height=1500)
-    st.write(dir(tokenizer))

 if text or button_clicked:
     st.write(len(tokenizer.tokenize(text)))
     st.components.v1.html(visualizer(text), height=1500)
+    st.write(tokenizer.vocab_size)