Cover Image

NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization

This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi.This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehe...

Full description

Saved in:
Bibliographic Details
Main Authors: Abu Bakar, Juhaida, Omar, Khairuddin, Nasrudin, Mohammad Faidzul, Murah, Mohd Zamri
Format: Article
Language:English
Published: Universiti Utara Malaysia 2016
Subjects:
PL Languages and literatures of Eastern Asia, Africa, Oceania
QA75 Electronic computers. Computer science
Online Access:http://repo.uum.edu.my/18485/1/JICT%2015%20%201%202016%20%20107%E2%80%93131.pdf
http://repo.uum.edu.my/18485/
http://www.jict.uum.edu.my/images/pdf3/vol15no1/51jict1512016.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
id my.uum.repo.18485
record_format eprints
spelling my.uum.repo.184852016-08-08T04:52:12Z http://repo.uum.edu.my/18485/ NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization Abu Bakar, Juhaida Omar, Khairuddin Nasrudin, Mohammad Faidzul Murah, Mohd Zamri PL Languages and literatures of Eastern Asia, Africa, Oceania QA75 Electronic computers. Computer science This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi.This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehensive, well-designed standard corpus that is annotated and made available for the public for the Jawi script corpora.This corpus contains the Jawi-specific Buckwalter character code and can be used to evaluate the performance of word tokenization tasks, as well as further language processing.The objective of this work is to conform and standardize the corpora between similar characters in Jawi.It consists of three subcorporas with documents from different genres. The gathering and processing steps, as well as the definition of several evaluation tasks regarding the use of these corpora, are included in this paper.One of the important roles and fundamental tasks of the corpus, which is the tokenization, is also presented in this paper.The development of the Malay language tokenizer is based on the syntactic data compatibility of Malay words written in Jawi.A series of experiments were performed to validate the corpus and to fulfill the requirement of the Jawi script tokenizer with an average error rate of 0.020255.Based on this promising result, the token will be used for the disambiguation and unknown word resolution, such as out-of vocabulary (OOV) problem in the tagging process. Universiti Utara Malaysia 2016-06 Article PeerReviewed application/pdf en http://repo.uum.edu.my/18485/1/JICT%2015%20%201%202016%20%20107%E2%80%93131.pdf Abu Bakar, Juhaida and Omar, Khairuddin and Nasrudin, Mohammad Faidzul and Murah, Mohd Zamri (2016) NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization. Journal of Information and Communication Technology (JICT), 15 (1). pp. 107-131. ISSN 1675-414X http://www.jict.uum.edu.my/images/pdf3/vol15no1/51jict1512016.pdf
institution Universiti Utara Malaysia
building UUM Library
collection Institutional Repository
continent Asia
country Malaysia
content_provider Universiti Utara Malaysia
content_source UUM Institutionali Repository
url_provider http://repo.uum.edu.my/
language English
topic PL Languages and literatures of Eastern Asia, Africa, Oceania
QA75 Electronic computers. Computer science
spellingShingle PL Languages and literatures of Eastern Asia, Africa, Oceania
QA75 Electronic computers. Computer science
Abu Bakar, Juhaida
Omar, Khairuddin
Nasrudin, Mohammad Faidzul
Murah, Mohd Zamri
NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization
description This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi.This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehensive, well-designed standard corpus that is annotated and made available for the public for the Jawi script corpora.This corpus contains the Jawi-specific Buckwalter character code and can be used to evaluate the performance of word tokenization tasks, as well as further language processing.The objective of this work is to conform and standardize the corpora between similar characters in Jawi.It consists of three subcorporas with documents from different genres. The gathering and processing steps, as well as the definition of several evaluation tasks regarding the use of these corpora, are included in this paper.One of the important roles and fundamental tasks of the corpus, which is the tokenization, is also presented in this paper.The development of the Malay language tokenizer is based on the syntactic data compatibility of Malay words written in Jawi.A series of experiments were performed to validate the corpus and to fulfill the requirement of the Jawi script tokenizer with an average error rate of 0.020255.Based on this promising result, the token will be used for the disambiguation and unknown word resolution, such as out-of vocabulary (OOV) problem in the tagging process.
format Article
author Abu Bakar, Juhaida
Omar, Khairuddin
Nasrudin, Mohammad Faidzul
Murah, Mohd Zamri
author_facet Abu Bakar, Juhaida
Omar, Khairuddin
Nasrudin, Mohammad Faidzul
Murah, Mohd Zamri
author_sort Abu Bakar, Juhaida
title NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization
title_short NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization
title_full NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization
title_fullStr NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization
title_full_unstemmed NUWT: Jawi-specific Buckwalter corpus for Malays word tokenization
title_sort nuwt: jawi-specific buckwalter corpus for malays word tokenization
publisher Universiti Utara Malaysia
publishDate 2016
url http://repo.uum.edu.my/18485/1/JICT%2015%20%201%202016%20%20107%E2%80%93131.pdf
http://repo.uum.edu.my/18485/
http://www.jict.uum.edu.my/images/pdf3/vol15no1/51jict1512016.pdf
_version_ 1644282466608873472
score 13.211869

Similar Items