- name
- Text
- description
- Transform, format, and process text with patterns for writing, data cleaning, localization, citations, and copywriting.
Quick Reference
| Task | Load |
|---|
| Creative writing (voice, dialogue, POV) | writing.md |
| Data processing (CSV, regex, encoding) | data.md |
| Academic/citations (APA, MLA, Chicago) | academic.md |
| Marketing copy (headlines, CTA, email) | copy.md |
| Translation/localization | localization.md |
Universal Text Rules
Encoding
- Always verify encoding first:
file -bi document.txt - Normalize line endings:
tr -d '\
' - Remove BOM if present:
sed -i '1s/^\ï\»\¿//'
Whitespace
- Collapse multiple spaces:
sed 's/[[:space:]]\+/ /g' - Trim leading/trailing:
sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
Common Traps
- Smart quotes (
" ") break parsers → normalize to " - Em/en dashes (
– —) break ASCII → normalize to - - Zero-width chars invisible but break comparisons → strip them
- String length ≠ byte length in UTF-8 (
"café" = 4 chars, 5 bytes)
Format Detection
# Detect encoding
file -I document.txt
# Detect line endings
cat -A document.txt | head -1
# ^M at end = Windows (CRLF)
# No ^M = Unix (LF)
# Detect delimiter (CSV/TSV)
head -1 file | tr -cd ',;\ |' | wc -c
Quick Transformations
| Task | Command |
|---|
| Lowercase | tr '[:upper:]' '[:lower:]' |
| Remove punctuation | tr -d '[:punct:]' |
| Count words | wc -w |
| Count unique lines | `sort -u \ | wc -l` |
| Find duplicates | `sort \ | uniq -d` |
| Extract emails | grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' |
| Extract URLs | `grep -oE 'https?://[^[:space:]<>"{} | \\^\[\]]+' |
Before Processing Checklist
- [ ] Encoding verified (UTF-8?)
- [ ] Line endings normalized
- [ ] Delimiter identified (for structured text)
- [ ] Target format/style defined
- [ ] Edge cases considered (empty, Unicode, special chars)
