If you don't have NLTK on your machine, go to http://www.nltk.org/download and install the latest version of NTLK
The CorpusReader script for the CHILDES corpus is part of the NLTK package, but the CHILDES corpus itself is not distributed with NLTK. You should download and save the CHILDES corpus.
Change the working directory to "nltk-read-only/nltk" and launch NLTK there.
If you run a Python script, they should be saved at "nltk-read-only/nltk" (or you can modify PYTHONPATH with export PYTHONPATH=~/ANY_PATH/nltk-read-only/nltk:$PYTHONPATH
# note thre is an 'nltk' dir inside of another 'nltk' dir
cd ~/ANY_PATH/nltk-read-only/nltk
python
>>> import nltk
# see if you are reading the correct version of the NLTK package
>>> nltk.__version__
>>> nltk.__file__
Changes on July 3, 2011 (thnaks to Alexis Dimitriadis for his suggestions)
The MLU and age methods can only look for the participant labeled CHI. But many transcripts (especially those involving more than one child) do not have a participant labeled CHI. So, these methods should have an optional parameter speaker='CHI' that they pass down, instead of the hard-coded CHI.
Other corpus readers have methods tagged_words and tagged_sents that do what childes does with the option pos=True. For consistency, I would propose adding wrapper functions with these names. They will be much easier to discover for students and new users.
I found it useful to access _convert_age( ) directly, but it is an undocumented internal function. Perhaps you could add it to the API? (By the way the try-except block is ineffective if there is no age: It will already fail on m.group(), if m == None).
For my students I created a very simple function that maps a childes xml file to the URL of the corresponding .cha file on the childes website, and opens it in a browser (using urllib).
A final thought: It would be nice if the speaker parameter would also accept a list, like the fileids parameter, so one can search all children, etc.
I presented a poster using the NLTK with the CHILDES corpus. The title of my poster (co-authored with Virgnia Valian) is "Is fully-automated corpus-based language acquisition research feasible?". A PDF version of the poster is avaialble below.
Testing your setup
If you have installed everything correctly, you should be able to run the following commands from your Terminal (on Mac OSX)
name@home:~/$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.corpus.reader.childes.demo()
Reading bates amy20 .....
words: ["what's", 'that', 'yyy', "it's", 'a', 'chicken', 'yeah'] ...
words with replaced words: ["what's", 'that', 'yyy', "it's", 'a', 'chicken', 'yeah'] ...
words with pos tags: [("what's", 'pro:wh'), ('that', 'pro:dem'), ('yyy', 'unk'), ("it's", 'pro:cop'), ('a', 'det'), ('chicken', 'n'), ('yeah', 'co')] ...
words (only MOT): ["what's", 'that', "it's", 'a', 'chicken', 'yeah', "what's"] ...
words (only CHI): ['yyy', 'yeah', 'yyy', 'yeah', 'xxx', 'woof', 'hat'] ...
stemmed words: ['what', 'be', 'that', 'yyy', 'it', 'be', 'a'] ...
words with relations and pos-tag: [[('what', 'pro:wh', '1|2|PRED'), ('be', 'v', '2|0|ROOT'), ('that', 'pro:dem', '3|2|SUBJ')], [('yyy', 'unk', '1|0|ROOT')], [('it', 'pro:cop', '1|2|SUBJ'), ('be', 'v', '2|0|ROOT'), ('a', 'det', '3|4|DET'), ('chicken', 'n', '4|2|PRED')], [('yeah', 'co', '1|0|ROOT')], [('yeah', 'co', '1|0|ROOT')]] ...
sentence: [["what's", 'that'], ['yyy']] ...
participant CHI group : normal
participant CHI language : eng
participant CHI age : P1Y8M
participant CHI sex : female
participant CHI role : Child
participant CHI id : CHI
participant MOT role : Mother
participant MOT id : MOT
participant MOT language : eng
num of sent: 170
num of morphemes: 296
age: ['P1Y8M']
age in month: [20]
MLU: [1.1428571428571428]
Reading bates betty20 .....
words: ['want', 'to', 'stack', 'them', 'for', 'me', 'yyy'] ...
words with replaced words: ['want', 'to', 'stack', 'them', 'for', 'me', 'yyy'] ...
words with pos tags: [('want', 'v'), ('to', 'inf'), ('stack', 'v'), ('them', 'pro'), ('for', 'prep'), ('me', 'pro'), ('yyy', 'unk')] ...
words (only MOT): ['want', 'to', 'stack', 'them', 'for', 'me', 'oh'] ...
words (only CHI): ['yyy', 'that', 'baby', 'yyy', 'yyy', 'uh', 'xxx'] ...
stemmed words: ['want', 'to', 'stack', 'them', 'for', 'me', 'yyy'] ...
words with relations and pos-tag: [[('want', 'v', '1|0|ROOT'), ('to', 'inf', '2|3|INF'), ('stack', 'v', '3|1|XCOMP'), ('them', 'pro', '4|3|OBJ'), ('for', 'prep', '5|3|JCT'), ('me', 'pro', '6|5|POBJ')], [], [], [('yyy', 'unk', '1|0|ROOT')], []] ...
sentence: [['want', 'to', 'stack', 'them', 'for', 'me'], []] ...
participant CHI group : normal
participant CHI name : Betty
participant CHI language : eng
participant CHI age : P1Y8M
participant CHI sex : female
participant CHI role : Target_Child
participant CHI id : CHI
participant MOT role : Mother
participant MOT id : MOT
participant MOT language : eng
num of sent: 148
num of morphemes: 406
age: ['P1Y8M']
age in month: [20]
MLU: [1.1428571428571428]
...
Basic usage
Read the CHILDESCorpusReader class and read the CHILDES corpus saved in the nltk_data directory.
name@home:~/$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> from nltk.corpus.reader import CHILDESCorpusReader
>>> corpus_root = nltk.data.find('corpora/childes/data-xml/Eng-USA/')
You can specify a subset of the corpora. For example, the following snippet reads files in the Valian corpus (Valian, 1991).
>>> corpus_data = valian.corpus(valian.fileids())
>>> print corpus_data[0]
{'Lang': 'eng', '{http://www.w3.org/2001/XMLSchema-instance}schemaLocation': ...
>>> for key in corpus_data[0].keys():
... print key, ": ", corpus_data[0][key]
Lang : eng
{http://www.w3.org/2001/XMLSchema-instance}schemaLocation : ...
Version : 1.5.6
Date : 1986-03-04
Corpus : valian
Id : 01a
Printing information of participants of the corpus. The most common codes for the participants are 'CHI' (target child), 'MOT' (mother), and 'INV' (investigator).
>>> corpus_participants = valian.participants(valian.fileids())
>>> for this_corpus_participants in corpus_participants[:2]:
... for key in this_corpus_participants.keys():
... print key, ": ", this_corpus_participants[key]
CHI : defaultdict(, {'group': 'normal'...
INV : defaultdict(, {'role': 'Investigator'...
MOT : defaultdict(, {'role': 'Mother',...
Printing words. You can specify the exact file path or use fileids().
>>> valian.words('Valian/01a.xml')
['at', 'Parent', "Lastname's", 'house', 'with', 'Child', 'Lastname', ...
>>> valian.words(valian.fileids()[0])
['at', 'Parent', "Lastname's", 'house', 'with', 'Child', 'Lastname', ...
# this will process all files in the Valian corpus
>>> valian.words(valian.fileids()[:])
['at', 'Parent', "Lastname's", 'house', 'with', 'Child', 'Lastname', ...
When the argument relation is true, the relational relationships in the sentence are returned. See Sagae et al. (2010) for details of the relational structure adopted in the CHILDES.
Printing MLU. The criteria for the MLU computation is broadly based on Brown (1973).
Note: The calculation formula for MLU is slightly different from one in CHAT (this should be fixed, but I don't have time for it). Therefore, MLUs calculated with NLTK might not match one by CHAT.
#---------- usingCHILDESWithNLTK2.py ----------#
# Description: Processing CHILDES XML data
# Date: January 17, 2011
# Written by Tomonori Nagano (tnagano@gc.cuny.edu) for Python2.5
import nltk
from nltk.probability import ConditionalFreqDist, FreqDist
from nltk.corpus.reader import CHILDESCorpusReader
modals = ["can","can't","could","couldn't","'ld","will","'11","won't","would","wouldn't","shall","should","shouldn't","may","might","must","ought"]
semimodal = ["gonna","wanna","hafta","gotta"]
corpus_root = nltk.data.find('corpora/childes/data-xml/Eng-USA/')
childes = CHILDESCorpusReader(corpus_root,u'.*.xml')
cfd = ConditionalFreqDist()
ages = ['1yld','2yld','3yld','4yld','5yld','6yld','7yld']
# change [:100] to [:] to run the entire CHILDES corpus
for filename in childes.fileids()[:]:
age = childes.age(filename,month=True)[0]
MLU = childes.MLU(filename)[0]
words = childes.words(filename,speaker=["CHI"])
if age is None:
pass
elif age < 12:
for word in words: cfd[ages[0]].inc(word)
elif age >= 12 and age < 24 :
for word in words: cfd[ages[1]].inc(word)
elif age >= 24 and age < 36 :
for word in words: cfd[ages[2]].inc(word)
elif age >= 36 and age < 48 :
for word in words: cfd[ages[3]].inc(word)
elif age >= 48 and age < 60 :
for word in words: cfd[ages[4]].inc(word)
elif age >= 60 and age < 72 :
for word in words: cfd[ages[5]].inc(word)
elif age >= 72 and age < 84 :
for word in words: cfd[ages[6]].inc(word)
print "age\tallTokens\t","\t".join(modals)
for thisAge in ages:
print thisAge,"\t",cfd[thisAge].N(),"\t",
for modal in modals:
print cfd[thisAge][modal],"\t",
print ""
print "age\tallTokens\t","\t".join(modals)
for thisAge in ages:
print thisAge,"\t",cfd[thisAge].N(),"\t",
for modal in modals:
try:
thisProp = float(cfd[thisAge][modal])/float(cfd[thisAge].N())*100
except:
thisProp = 0
print "%.2f%%\t" % thisProp,
print ""
Example 4 (Checking word distributions of adult and child utterances)
#---------- usingCHILDESWithNLTK4.py ----------#
# Description: Processing CHILDES XML data
# Date: February 6, 2011
# Written by Tomonori Nagano (tnagano@gc.cuny.edu) for Python2.5
import sys
sys.path.append('/Library/Python/2.6/site-packages/')
import nltk
from nltk.probability import ConditionalFreqDist, FreqDist
from nltk.corpus.reader import CHILDESCorpusReader
import random
corpus_root = nltk.data.find('corpora/CHILDES/data-xml/Eng-USA/')
childes = CHILDESCorpusReader(corpus_root,u'.*.xml')
sampleNum = 3000
child_cfd = ConditionalFreqDist()
adult_cfd = ConditionalFreqDist()
ages = ['1yld','2yld','3yld','4yld','5yld','6yld','7yld']
def make_cfd(ageKey):
if child_cfd[ages[ageKey]].N() > sampleNum:
pass
else:
words = childes.words(filename,speaker="CHI")
for word in words: child_cfd[ages[ageKey]].inc(word)
if adult_cfd[ages[ageKey]].N() > sampleNum:
pass
else:
words = childes.words(filename,speaker="MOT")
for word in words: adult_cfd[ages[ageKey]].inc(word)
# for filename in random.sample(childes.fileids(),100*7):
for filename in random.sample(childes.fileids(),500):
age = childes.age(filename,month=True)[0]
if age is None:
pass
elif age < 12:
make_cfd(0)
elif age >= 12 and age < 24 :
make_cfd(1)
elif age >= 24 and age < 36 :
make_cfd(2)
elif age >= 36 and age < 48 :
make_cfd(3)
elif age >= 48 and age < 60 :
make_cfd(4)
elif age >= 60 and age < 72 :
make_cfd(5)
elif age >= 72 and age < 84 :
make_cfd(6)
for age in ages:
filename = "child"+age+".wfl"
outfile = open(filename,"w")
outfile.write("type\tf\r")
for word in child_cfd[age].keys():
outfile.write(unicode(word).encode("utf-8")+"\t"+str(child_cfd[age][word])+"\r")
outfile.close()
for age in ages:
filename = "adult"+age+".wfl"
outfile = open(filename,"w")
outfile.write("type\tf\r")
for word in adult_cfd[age].keys():
outfile.write(unicode(word).encode("utf-8")+"\t"+str(adult_cfd[age][word])+"\r")
outfile.close()
################################################
## R procedures
################################################
## clear the cache
#rm(list = ls())
#
## change the default width
#width.default <- getOption("width"); options(width=90)
#
#library(zipfR)
#setwd("~/Desktop/")
#
#colors = topo.colors(10, alpha = 0.5)
#speakers = c("adult","child")
#ages = c('1yld','2yld','3yld','4yld','5yld','6yld','7yld')
#for (j in 1:length(speakers)){
# for (i in 1:length(ages)){
# pdf(paste(speakers[j],"_spc_",ages[i],".pdf",sep=""), width = 8, height = 6, onefile = TRUE, pointsize = 9)
# filename <- paste(speakers[j],ages[i],".wfl",sep="")
# this_spc <- tfl2spc(read.tfl(filename))
# this_lnre <- lnre("zm",this_spc)
# this_text <- paste("LNRE Model\n--------------------\n","Method: ",this_lnre$name,"\n","alpha=",format(this_lnre$param$alpha,digits=3)," B=",format(this_lnre$param$B,digits=3),"\n","Goodness-of-fit\n","X^2(df=",this_lnre$gof$df,")=",format(this_lnre$gof$X2,digits=1),", p=",format(this_lnre$gof$p,digits=1),sep="")
# plot(this_spc,main=paste("Frequency spectrum of ",speakers[j]," utterances in CHILDES\n(",ages[i],"; n=3000; randomly samplled)\nPrinted on: ",format(Sys.time(), "%b %d, %Y"),sep=""),xlab="Frequency Class, m (m=1 is hapex legomenon)",ylab="Frequency of Frequency Class, V_m",barcol=colors[i])
# mtext(this_text,side=3,adj=1,padj=2)
# dev.off()
#
# pdf(paste(speakers[j],"_vgc_",ages[i],".pdf",sep=""), width = 8, height = 6, onefile = TRUE, pointsize = 9)
# this_vgc <- vgc.interp(this_spc,(1:100)*N(this_spc)/100)
# plot(this_vgc,main=paste("Vocabulary growth of ",speakers[j]," utterances in CHILDES\n(",ages[i],"; n=3000; randomly samplled)\nPrinted on: ",format(Sys.time(), "%b %d, %Y"),sep=""),xlab="Number of words, N",ylab="Binomially interpolated vocabulary growth curve, E[V(N)]",col=colors[i])
# dev.off()
# }
#}
#
