PAMI.periodicFrequentPattern.pyspark package

Submodules

PAMI.periodicFrequentPattern.pyspark.abstract module

PAMI.periodicFrequentPattern.pyspark.parallelPFPGrowth module

class PAMI.periodicFrequentPattern.pyspark.parallelPFPGrowth.Node(item, count, children)

Bases: object

A class used to represent the node of frequentPatternTree

Attributes:

itemint or None

Storing item of a node

timeStampslist

To maintain the timestamps of a database at the end of the branch

parentnode

To maintain the parent of every node

childrenlist

To maintain the children of a node

countint

To maintain the count of every node

Methods:

addChild(itemName)

Storing the children to their respective parent nodes

toString()

To print the node

addChild(node)

To add the children to a node

Parameters:

node – children of a node

toString(level=0)

To print the node

Parameters:

level – level of a node

class PAMI.periodicFrequentPattern.pyspark.parallelPFPGrowth.PFPTree

Bases: object

A class used to represent the periodic frequent pattern tree

Attributes:

rootnode

To maintain the root of the tree

summariesdict

To maintain the summary of the tree

Methods:

add(basket, tid, count)

To add the basket to the tree

getTransactions()

To get the transactions of the tree

merge(tree)

To merge the tree

project(itemId)

To project the tree

satisfyPer(tids, maxPer, numTrans)

To satisfy the periodicity constraint

extract(minCount, maxPer, numTrans, isResponsible = lambda x:True)

To extract the periodic frequent patterns

add(basket, tid, count)

To add the basket to the tree

Parameters:

• basket – basket of a database

• tid – timestamp of a database

• count – count of a node

extract(minCount, maxPer, numTrans, isResponsible=<function PFPTree.<lambda>>)

To extract the periodic frequent patterns

Parameters:

• minCount – minimum count of a node

• maxPer – maximum periodicity

• numTrans – number of transactions

• isResponsible – responsible node of a tree

getTransactions()

To get the transactions of the tree

Returns:

returning the transactions of the tree

merge(tree)

To merge the tree

Parameters:

tree – tree of a database

project(itemId)

To project the tree

Parameters:

itemId – item of a node

satisfyPer(tids, maxPer, numTrans)

To satisfy the periodicity constraint

Parameters:

• tids – timestamps of a database

• maxPer – maximum periodicity

• numTrans – number of transactions

class PAMI.periodicFrequentPattern.pyspark.parallelPFPGrowth.Summary(count, nodes)

Bases: object

A class used to represent the summary of the tree

Attributes:

countint

To maintain the count of a node

nodeslist

To maintain the nodes of a tree

tidsset

To maintain the timestamps of a database

class PAMI.periodicFrequentPattern.pyspark.parallelPFPGrowth.parallelPFPGrowth(iFile, minSup, maxPer, numWorker, sep='\t')

Bases: _periodicFrequentPatterns

Description:

ParallelPFPGrowth is one of the fundamental distributed algorithm to discover periodic-frequent patterns in a transactional database. It is based PySpark framework.

Reference:

3. Saideep, R. Uday Kiran, Koji Zettsu, Cheng-Wei Wu, P. Krishna Reddy, Masashi Toyoda, Masaru Kitsuregawa: Parallel Mining of Partial Periodic Itemsets in Big Data. IEA/AIE 2020: 807-819

param iFile:

str : Name of the Input file to mine complete set of periodic frequent pattern’s

param oFile:

str : Name of the output file to store complete set of periodic frequent pattern’s

param minSup:

str: Controls the minimum number of transactions in which every item must appear in a database.

param maxPer:

str: Controls the maximum number of transactions in which any two items within a pattern can reappear.

param sep:

str : This variable is used to distinguish items from one another in a transaction. The default seperator is tab space. However, the users can override their default separator.

Attributes:

iFilefile

Name of the Input file or path of the input file

oFilefile

Name of the output file or path of the output file

minSup: int or float or str

The user can specify minSup either in count or proportion of database size. If the program detects the data type of minSup is integer, then it treats minSup is expressed in count. Otherwise, it will be treated as float. Example: minSup=10 will be treated as integer, while minSup=10.0 will be treated as float

maxPer: int or float or str

The user can specify maxPer either in count or proportion of database size. If the program detects the data type of maxPer is integer, then it treats maxPer is expressed in count. Otherwise, it will be treated as float. Example: maxPer=10 will be treated as integer, while maxPer=10.0 will be treated as float

numWorker: int

The user can specify the number of worker machines to be employed for finding periodic-frequent patterns.

sepstr

This variable is used to distinguish items from one another in a transaction. The default seperator is tab space or . However, the users can override their default separator.

memoryUSSfloat

To store the total amount of USS memory consumed by the program

memoryRSSfloat

To store the total amount of RSS memory consumed by the program

startTime:float

To record the start time of the mining process

endTime:float

To record the completion time of the mining process

Databaselist

To store the transactions of a database in list

mapSupportDictionary

To maintain the information of item and their frequency

lnoint

To represent the total no of transaction

treeclass

To represents the Tree class

itemSetCountint

To represents the total no of patterns

finalPatternsdict

To store the complete patterns

Methods:

mine()

Mining process will start from here

getPatterns()

Complete set of patterns will be retrieved with this function

save(oFile)

Complete set of periodic-frequent patterns will be loaded in to a output file

getPatternsAsDataFrame()

Complete set of periodic-frequent patterns will be loaded in to a dataframe

getMemoryUSS()

Total amount of USS memory consumed by the mining process will be retrieved from this function

getMemoryRSS()

Total amount of RSS memory consumed by the mining process will be retrieved from this function

getRuntime()

Total amount of runtime taken by the mining process will be retrieved from this function

creatingItemSets(fileName)

Scans the dataset and stores in a list format

PeriodicFrequentOneItem()

Extracts the one-periodic-frequent patterns from database

updateDatabases()

Update the database by removing aperiodic items and sort the Database by item decreased support

buildTree()

After updating the Database, remaining items will be added into the tree by setting root node as null

convert()

to convert the user specified value

   Format:


   (.venv) $ python3 parallelPFPGrowth.py <inputFile> <outputFile> <minSup> <maxPer> <noWorker>

   Example usage :

   (.venv) $ python3 parallelPFPGrowth.py sampleTDB.txt patterns.txt 0.3 0.4 5


           .. note:: minSup will be considered in percentage of database transactions


**Importing this algorithm into a python program**
-----------------------------------------------------
.. code-block:: python

            from PAMI.periodicFrequentPattern.basic import parallelPFPGrowth as alg

            obj = alg.parallelPFPGrowth(iFile, minSup, maxPer)

            obj.mine()

            periodicFrequentPatterns = obj.getPatterns()

            print("Total number of Periodic Frequent Patterns:", len(periodicFrequentPatterns))

            obj.savePatterns(oFile)

            Df = obj.getPatternsAsDataFrame()

            memUSS = obj.getMemoryUSS()

            print("Total Memory in USS:", memUSS)

            memRSS = obj.getMemoryRSS()

            print("Total Memory in RSS", memRSS)

            run = obj.getRuntime()

            print("Total ExecutionTime in seconds:", run)

Mine()

Start the mining process

func1(ps1, tid)

Add the tid to the set

Parameters:

• ps1 – set

• tid – timestamp of a database

return: set

func2(ps1, ps2)

Union of two sets

Parameters:

• ps1 – set.

• ps2 – set

return: set

func3(tids, endts)

Calculate the periodicity of a transaction

Parameters:

tids – timestamps of a database

return: periodicity

genCondTransactions(tid, basket, rank, nPartitions)

Get the conditional transactions from the database

Parameters:

• tid – timestamp of a database

• basket – basket of a database

• rank – rank of a database

• nPartitions – number of partitions

getFrequentItems(data)

Get the frequent items from the database

Parameters:

data – database

return: frequent items

getFrequentItemsets(data, freqItems)

Get the frequent itemsets from the database

Parameters:

• data – database

• freqItems – frequent items

return: frequent itemsets

getMemoryRSS()

Total amount of RSS memory consumed by the mining process will be retrieved from this function

Returns:

returning RSS memory consumed by the mining process

Return type:

float

getMemoryUSS()

Total amount of USS memory consumed by the mining process will be retrieved from this function

Returns:

returning USS memory consumed by the mining process

Return type:

float

getPartitionId(key, nPartitions)

Get the partition id

Parameters:

• key – key of a database

• nPartitions – number of partitions

return: partition id

getPatterns()

Function to send the set of frequent patterns after completion of the mining process

Returns:

returning frequent patterns

Return type:

dict

getPatternsAsDataFrame()

Storing final frequent patterns in a dataframe

Returns:

returning frequent patterns in a dataframe

Return type:

pd.DataFrame

getRuntime()

Calculating the total amount of runtime taken by the mining process

Returns:

returning total amount of runtime taken by the mining process

Return type:

float

printResults()

To print results of the execution.

save(outFile)

Complete set of frequent patterns will be loaded in to a output file

Parameters:

outFile (csv file) – name of the output file

startMine()

Start the mining process

Module contents