The corpus is comprised of HTML and XML files. Language identification was performed on all threads in this corpus (using CLD2), and threads for which the results indicated a high probability of largely non-Chinese content are listed in cmn_suspect_LID.txt in the docs directory of this package. Only a small portion of the threads included in this release were manually reviewed, and it is expected that there may be some offensive or otherwise undesired content as well as some threads that contain a large amount of non-Chinese content. The scale of the collection precluded manual review of all data. When multiple threads from a forum were submitted, the entire forum was automatically harvested and added to the collection. Upon locating an appropriate thread, scouts submitted the URL and some simple judgments about it to a database, via a web browser plug-in. ![]() Scouts were instructed to seek content in Mandarin Chinese that was original, interactive and informal. DataĬollection was seeded based on the results of manual data scouting by native speaker annotators. The data was subseqently translated and annotated for various tasks in the BOLT program including word alignment, treebanking, propbanking and co-reference. The material in this release represents the unannotated Chinese source data in the discussion forum genre. LDC supported the BOLT program by collecting informal data sources - discussion forums, text messaging and chat - in Chinese, Egyptian Arabic and English. ![]() The DARPA BOLT (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. BOLT Chinese Discussion Forums was developed by the Linguistic Data Consortium (LDC) and consists of 1,597,500 discussion forum threads in Chinese harvested from the Internet using a combination of manual and automatic processes.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |