Blogpie474

Latest version

If you’re unsure of which datasets/models you’ll need, you can install the “popular” subset of NLTK data, on the command line type python -m nltk.downloader popular, or in the Python interpreter import nltk; nltk.download(‘popular’). VPython makes it easy to create navigable 3D displays and animations, even for those with limited programming experience. Because it is based on Python, it also has much to offer for experienced programmers and researchers. Click the 'Run' button above to see a 3D animation.

Released:

A library for working with Adblock Plus filter lists.

Project description

This repository contains a library for working with Adblock Plus filter lists,a script for rendering diffs between filter lists, and the script that is usedfor building Adblock Plus filter lists from the form in which they are authoredinto the format suitable for consumption by the adblocking software (akarendering).

Contents

• Library API
• Using the library with R

Installation

Prerequisites:

• Linux, Mac OS X or Windows (any modern Unix should work too),
• Python (2.7 or 3.5+),
• pip.

To install:

Rendering of filter lists

The filter lists are originally authored in relatively smaller parts focusedon particular types of filters, related to a specific topic or relevant for aparticular geographical area.We call these parts filter list fragments (or just fragments) todistinguish them from full filter lists that are consumed by the adblockingsoftware such as Adblock Plus.

Rendering is a process that combines filter list fragments into a filter list.It starts with one fragment that can include other ones and so forth.The produced filter list is marked with a version and a timestamp.

Python-abp contains a script that can do this called flrender:

This will take the top level fragment in fragment.txt, render it and save itinto filterlist.txt.

Download Python 3.7 Mac

The flrender script can also be used by only specifying fragment.txt:

in which case the rendering result will be sent to stdout. Moreover, whenit’s run with no positional arguments:

it will read from stdin and send the results to stdout.

Fragments might reference other fragments that should be included into them.The references come in two forms: http(s) includes and local includes:

The http include contains a URL that will be fetched and inserted at the pointof reference.The local include contains a path inside the easylist repository.flrender needs to be able to find a copy of the repository on the localfilesystem. We use -i option to point it to to the right directory:

Now the local include referenced above will be resolved to:/home/abc/easylist/easylist/easylist_general_block.txtand the fragment will be loaded from this file.

Directories that contain filter list fragments that are used during renderingare called sources.They are normally working copies of the repositories that contain filter listfragments.Each source is identified by a name: that’s the part that comes before “:” inthe include instruction and it should be the same as what comes before “=” inthe -i option.

Commonly used sources have generally accepted names. For example the mainEasyList repository is referred to as easylist.If you don’t know all the source names that are needed to render some list,just run flrender and it will report what it’s missing:

You can clone the necessary repositories to a local directory and add -ioptions accordingly.

Generating diffs

A diff allows a client running ad blocking software such as Adblock Plus toupdate the filter lists incrementally, instead of downloading a new copy of afull list during each update. This is meant to lessen the amount of resourcesused when updating filter lists (e.g. network data, memory usage, batteryconsumption, etc.), allowing clients to update their lists more frequentlyusing less resources.

python-abp contains a script called fldiff that will find the diff betweenthe latest filter list, and any number of previous filter lists:

where -o diffs/easylist/ is the (optional) output directory where the diffsshould be written, easylist.txt is the most recent version of the filterlist, and archive/* is the directory where all the archived filter lists are.When called like this, the shell should automatically expand the archive/*directory, giving the script each of the filenames separately.

In the above example, the output of each archived list[version].txt will bewritten to diffs/diff[version].txt. If the output argument is omitted, thediffs will be written to the current directory.

The script produces three types of lines, as specified in the technicalspecification:

• Special comments of the form ! <name>:[ <value>]
• Added filters of the form + <filter-text>
• Removed filters of the form - <filter-text>

Library API

python-abp can also be used as a library for parsing filter lists. For exampleto read a filter list (we use Python 3 syntax here but the API is the same):

If filterlist.txt contains this filter list:

the output will look something like:

The abp.filters module also exports a lower-level function for parsingindividual lines of a filter list: parse_line. It returns a parsed lineobject just like the items in the iterator returned by parse_filterlist.

For further information on the library API use help() on abp.filters andits contents in an interactive Python session, read the docstrings, or look atthe tests for some usage examples.

Blocks of filters

Further processing of blocks of filters separated by comments can be performedusing to_blocks function from abp.filters.blocks:

Use help() on abp.filters.blocks for more information.

Testing

Unit tests for python-abp are located in the /tests directory. Pytestis used for quickly running the tests during development. Tox is used fortesting in different environments (Python 2.7, Python 3.5+ and PyPy) and codequality reporting.

Use tox for a comprehensive report of unit tests and test coverage:

Development

When adding new functionality, add tests for it (preferably first). If somecode will never be reached on a certain version of Python, it may be exemptedfrom coverage tests by adding a comment, e.g. # pragma: no py2 cover.

All public functions, classes and methods should have docstrings compliant withNumPy/SciPy documentation guide.One exception is the constructors of classes that the user is not expected toinstantiate (such as exceptions).

Using the library with R

Installation

python-abp can be installed from PyPI or from the source code, eitherdirectly onto a system or in a virtual environment.

To install from PyPI:

To install from a local source, clone the repo and then:

To use the virtual environment, it must first be created. Python 2 and 3 usedifferent scripts to create a virtualenv.

In Python 2:

In Python 3:

Then, use the virtualenv’s version of pip to install python-abp, either fromPyPI or from source (as shown above):

For more information about virtualenv, please see the User Guide and thedocs.

Usage

In R, python-abp can be imported with reticulate:

Now you can use the functions with abp$functionname, e.g.abp$line2dict('@@||g.doubleclick.net/pagead/$subdocument,domain=hon30.org')

For more information about the reticulate package, see their guide.

Release historyRelease notifications | RSS feed

0.2.0

0.1.3

0.1.2

0.1.1

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Hashes for python-abp-0.2.0.tar.gz

Released:

A library for working with Adblock Plus filter lists.

Project description

This repository contains a library for working with Adblock Plus filter lists,a script for rendering diffs between filter lists, and the script that is usedfor building Adblock Plus filter lists from the form in which they are authoredinto the format suitable for consumption by the adblocking software (akarendering).

Contents

• Library API
• Using the library with R

Installation

Prerequisites:

• Linux, Mac OS X or Windows (any modern Unix should work too),
• Python (2.7 or 3.5+),
• pip.

To install:

Rendering of filter lists

The filter lists are originally authored in relatively smaller parts focusedon particular types of filters, related to a specific topic or relevant for aparticular geographical area.We call these parts filter list fragments (or just fragments) todistinguish them from full filter lists that are consumed by the adblockingsoftware such as Adblock Plus.

Rendering is a process that combines filter list fragments into a filter list.It starts with one fragment that can include other ones and so forth.The produced filter list is marked with a version and a timestamp.

Python-abp contains a script that can do this called flrender:

This will take the top level fragment in fragment.txt, render it and save itinto filterlist.txt.

The flrender script can also be used by only specifying fragment.txt:

in which case the rendering result will be sent to stdout. Moreover, whenit’s run with no positional arguments:

it will read from stdin and send the results to stdout.

Fragments might reference other fragments that should be included into them.The references come in two forms: http(s) includes and local includes:

The http include contains a URL that will be fetched and inserted at the pointof reference.The local include contains a path inside the easylist repository.flrender needs to be able to find a copy of the repository on the localfilesystem. We use -i option to point it to to the right directory:

Now the local include referenced above will be resolved to:/home/abc/easylist/easylist/easylist_general_block.txtand the fragment will be loaded from this file.

Directories that contain filter list fragments that are used during renderingare called sources.They are normally working copies of the repositories that contain filter listfragments.Each source is identified by a name: that’s the part that comes before “:” inthe include instruction and it should be the same as what comes before “=” inthe -i option.

Commonly used sources have generally accepted names. For example the mainEasyList repository is referred to as easylist.If you don’t know all the source names that are needed to render some list,just run flrender and it will report what it’s missing:

You can clone the necessary repositories to a local directory and add -ioptions accordingly.

Generating diffs

A diff allows a client running ad blocking software such as Adblock Plus toupdate the filter lists incrementally, instead of downloading a new copy of afull list during each update. This is meant to lessen the amount of resourcesused when updating filter lists (e.g. network data, memory usage, batteryconsumption, etc.), allowing clients to update their lists more frequentlyusing less resources.

python-abp contains a script called fldiff that will find the diff betweenthe latest filter list, and any number of previous filter lists:

where -o diffs/easylist/ is the (optional) output directory where the diffsshould be written, easylist.txt is the most recent version of the filterlist, and archive/* is the directory where all the archived filter lists are.When called like this, the shell should automatically expand the archive/*directory, giving the script each of the filenames separately.

In the above example, the output of each archived list[version].txt will bewritten to diffs/diff[version].txt. If the output argument is omitted, thediffs will be written to the current directory.

The script produces three types of lines, as specified in the technicalspecification:

• Special comments of the form ! <name>:[ <value>]
• Added filters of the form + <filter-text>
• Removed filters of the form - <filter-text>

Library API

python-abp can also be used as a library for parsing filter lists. For exampleto read a filter list (we use Python 3 syntax here but the API is the same):

If filterlist.txt contains this filter list:

the output will look something like:

The abp.filters module also exports a lower-level function for parsingindividual lines of a filter list: parse_line. It returns a parsed lineobject just like the items in the iterator returned by parse_filterlist.

For further information on the library API use help() on abp.filters andits contents in an interactive Python session, read the docstrings, or look atthe tests for some usage examples.

Blocks of filters

Further processing of blocks of filters separated by comments can be performedusing to_blocks function from abp.filters.blocks:

Use help() on abp.filters.blocks for more information.

Testing

Unit tests for python-abp are located in the /tests directory. Pytestis used for quickly running the tests during development. Tox is used fortesting in different environments (Python 2.7, Python 3.5+ and PyPy) and codequality reporting.

Use tox for a comprehensive report of unit tests and test coverage:

Development

Download Vpython Module Mac For Python 3.5 Full

When adding new functionality, add tests for it (preferably first). If somecode will never be reached on a certain version of Python, it may be exemptedfrom coverage tests by adding a comment, e.g. # pragma: no py2 cover.

All public functions, classes and methods should have docstrings compliant withNumPy/SciPy documentation guide.One exception is the constructors of classes that the user is not expected toinstantiate (such as exceptions).

Using the library with R

Installation

python-abp can be installed from PyPI or from the source code, eitherdirectly onto a system or in a virtual environment.

Python Program

To install from PyPI:

To install from a local source, clone the repo and then:

To use the virtual environment, it must first be created. Python 2 and 3 usedifferent scripts to create a virtualenv.

In Python 2:

In Python 3:

Then, use the virtualenv’s version of pip to install python-abp, either fromPyPI or from source (as shown above):

For more information about virtualenv, please see the User Guide and thedocs.

Usage

In R, python-abp can be imported with reticulate:

Now you can use the functions with abp$functionname, e.g.abp$line2dict('@@||g.doubleclick.net/pagead/$subdocument,domain=hon30.org')

For more information about the reticulate package, see their guide.

Download Python 3 For Mac

Release historyRelease notifications | RSS feed

0.2.0

0.1.3

0.1.2

0.1.1

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Download Vpython Module Mac For Python 3.5 Free

Hashes for python-abp-0.2.0.tar.gz