tim1900 commited on
Commit
49450c6
·
verified ·
1 Parent(s): 552f53f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -1
README.md CHANGED
@@ -132,7 +132,9 @@ Three types of constraints may be specified in disciplined convex programs:
132
  _Non_-equality constraints, constructed using \(\sim=\), are never allowed. (Such constraints are not convex.)
133
 
134
  One or both sides of an equality constraint may be complex; inequality constraints, on the other hand, must be real. A complex equality constraint is equivalent to two real equality constraints, one for the real part and one for the imaginary part. An equality constraint with a real side and a complex side has the effect of constraining the imaginary part of the complex side to be zero."""
135
- # chunk the text. The prob_threshold should be between (0, 1). The lower it is, the more chunks will be generated.
 
 
136
  chunks, token_pos = chunk_text(model, doc, tokenizer, prob_threshold=0.5)
137
 
138
  # print chunks
 
132
  _Non_-equality constraints, constructed using \(\sim=\), are never allowed. (Such constraints are not convex.)
133
 
134
  One or both sides of an equality constraint may be complex; inequality constraints, on the other hand, must be real. A complex equality constraint is equivalent to two real equality constraints, one for the real part and one for the imaginary part. An equality constraint with a real side and a complex side has the effect of constraining the imaginary part of the complex side to be zero."""
135
+ # Chunk the text. The prob_threshold should be between (0, 1). The lower it is, the more chunks will be generated.
136
+ # Therefore adjust it to your need, when prob_threshold is small like 0.000001, each token is one chunk,
137
+ # when it is set to 1, the whole text will be one chunk.
138
  chunks, token_pos = chunk_text(model, doc, tokenizer, prob_threshold=0.5)
139
 
140
  # print chunks