Update README.md
Browse files
README.md
CHANGED
|
@@ -132,7 +132,9 @@ Three types of constraints may be specified in disciplined convex programs:
|
|
| 132 |
_Non_-equality constraints, constructed using \(\sim=\), are never allowed. (Such constraints are not convex.)
|
| 133 |
|
| 134 |
One or both sides of an equality constraint may be complex; inequality constraints, on the other hand, must be real. A complex equality constraint is equivalent to two real equality constraints, one for the real part and one for the imaginary part. An equality constraint with a real side and a complex side has the effect of constraining the imaginary part of the complex side to be zero."""
|
| 135 |
-
#
|
|
|
|
|
|
|
| 136 |
chunks, token_pos = chunk_text(model, doc, tokenizer, prob_threshold=0.5)
|
| 137 |
|
| 138 |
# print chunks
|
|
|
|
| 132 |
_Non_-equality constraints, constructed using \(\sim=\), are never allowed. (Such constraints are not convex.)
|
| 133 |
|
| 134 |
One or both sides of an equality constraint may be complex; inequality constraints, on the other hand, must be real. A complex equality constraint is equivalent to two real equality constraints, one for the real part and one for the imaginary part. An equality constraint with a real side and a complex side has the effect of constraining the imaginary part of the complex side to be zero."""
|
| 135 |
+
# Chunk the text. The prob_threshold should be between (0, 1). The lower it is, the more chunks will be generated.
|
| 136 |
+
# Therefore adjust it to your need, when prob_threshold is small like 0.000001, each token is one chunk,
|
| 137 |
+
# when it is set to 1, the whole text will be one chunk.
|
| 138 |
chunks, token_pos = chunk_text(model, doc, tokenizer, prob_threshold=0.5)
|
| 139 |
|
| 140 |
# print chunks
|