Dark Mode

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

mscaudill/tabbed

Repository files navigation

A Python package for reading variably structured text files at scale

Tabbed is a Python library for reading variably structured text files. It automatically deduces data start locations, data types and performs iterative and value-based conditional reading of data rows.

Key Features | Usage | Documentation | Dependencies | Installation | Contributing | Acknowledgments

Key Features

  • Structural Inference:
    A common variant of the standard text file is one that contains metadata prior to a header or data section. Tabbed can locate the metadata, header and data locations in a file.

  • Type inference:
    Tabbed can parse int, float, complex, time, date and datetime instances at high-speed via a polling strategy.

  • Conditional Reading:
    Tabbed can filter rows during reading with equality, membership, rich comparison, regular expression matching and custom callables via simple keyword arguments.

  • Partial and Iterative Reading:
    Tabbed supports reading of large text files that consumes only as much memory as you choose.

Usage

Below is a sample file with a Metadata section and Header using the tab character as the delimiter.

annotations.txt

Experiment ID Experiment
Animal ID Animal
Researcher Test
Directory path

Number Start Time End Time Time From Start Channel Annotation
0 02/09/22 09:17:38.948 02/09/22 09:17:38.948 0.0000 ALL Started Recording
1 02/09/22 09:37:00.000 02/09/22 09:37:00.000 1161.0520 ALL start
2 02/09/22 09:37:00.000 02/09/22 09:37:08.784 1161.0520 ALL exploring
3 02/09/22 09:37:08.784 02/09/22 09:37:13.897 1169.8360 ALL grooming
4 02/09/22 09:37:13.897 02/09/22 09:38:01.262 1174.9490 ALL exploring
5 02/09/22 09:38:01.262 02/09/22 09:38:07.909 1222.3140 ALL grooming
6 02/09/22 09:38:07.909 02/09/22 09:38:20.258 1228.9610 ALL exploring
7 02/09/22 09:38:20.258 02/09/22 09:38:25.435 1241.3100 ALL grooming
8 02/09/22 09:38:25.435 02/09/22 09:40:07.055 1246.4870 ALL exploring
9 02/09/22 09:40:07.055 02/09/22 09:40:22.334 1348.1070 ALL grooming
10 02/09/22 09:40:22.334 02/09/22 09:41:36.664 1363.3860 ALL exploring

Dialect and Type Inference

Tabbed can detect the dialect via clevercsv and infer the data types.

from tabbed.reading import Reader
from tabbed.samples import paths

infile = open(paths.annotations, 'r')
reader = Reader(infile)
dialect = reader.sniffer.dialect
types, _ = reader.sniffer.types(poll=10)

print(dialect) # a clevercsv SimpleDialect
print('---')
print(types)

Output

, , , , , ]">SimpleDialect('\t', '"', None)
---
[, , , , , ]

Metadata and Header detection

Tabbed can automatically locate the metadata, header and data rows.

print(reader.header)
print('---')
print(reader.metadata())

Output

Header(line=6,
names=['Number', 'Start_Time', 'End_Time', 'Time_From_Start', 'Channel', 'Annotation'],
string='Number\tStart Time\tEnd Time\tTime From Start\tChannel\tAnnotation')
---
MetaData(lines=(0, 6),
string='Experiment ID\tExperiment\nAnimal ID\tAnimal\nResearcher\tTest\nDirectory path\t\n\n')

Filtered Reading with Tabs

Tabbed supports row and column filtering with equality, membership, rich comparison and regular expression matching. Its also fully iterative allowing users to choose the amount of memory to consume during file reading.

= 2/09/2022 9:38:00 and <2/09/2022 9:40:00', columns=['Number', 'Start_Time'] ) # read the data to an iterator reading only 2 rows at a time gen = reader.read(chunksize=2) # convert to an in-memory list data = list(chain.from_iterable(gen)) print(data) # close the reader when done or open under context-management reader.close()">from itertools import chain

# tab rows whose Start_Time is between 9:38 and 9:40 and set reader to read
# only the Number and Start_Time columns
reader.tab(
Start_Time='>= 2/09/2022 9:38:00 and <2/09/2022 9:40:00',
columns=['Number', 'Start_Time']
)

# read the data to an iterator reading only 2 rows at a time
gen = reader.read(chunksize=2)

# convert to an in-memory list
data = list(chain.from_iterable(gen))
print(data)

# close the reader when done or open under context-management
reader.close()

Output

{'Number': 5, 'Start_Time': datetime.datetime(2022, 2, 9, 9, 38, 1, 262000)}
{'Number': 6, 'Start_Time': datetime.datetime(2022, 2, 9, 9, 38, 7, 909000)}
{'Number': 7, 'Start_Time': datetime.datetime(2022, 2, 9, 9, 38, 20, 258000)}
{'Number': 8, 'Start_Time': datetime.datetime(2022, 2, 9, 9, 38, 25, 435000)}

Documentation

The official documentation is hosted on github.io.

Dependencies

Tabbed depends on the excellent clevercsv package for dialect detection. The rest is pure Python.

Installation

Tabbed is hosted on pypi and can be installed with pip into a virtual environment.

pip install tabbed

To get a development version of Tabbed from source start by cloning the repository

git clone git@github.com:mscaudill/tabbed.git

Go to the directory you just cloned and create an editable install with pip.

pip install -e .[dev]

Contributing

We're excited you want to contribute! Please check out our Contribution guide.

Acknowledgements

We are grateful for the support of the Ting Tsung and Wei Fong Chao Foundation and the Jan and Dan Duncan Neurological Research Institute at Texas Children's that generously supports Tabbed.

About

A Python package for reading variably structured text files at scale

Topics

Resources

Readme

License

BSD-3-Clause license

Code of conduct

Code of conduct

Contributing

Contributing

Stars

Watchers

Forks

Packages

Contributors