Jeffrey Beall’s blog listing of potential predatory journals and publishers, as well as his Criteria for Determining Predatory Open-Access (OA) Publishers are often looked at as tools to help researchers avoid publishing in predatory journals. While these Criteria has brought a greater awareness of OA predatory journals, these tools alone should not be used as the only source in determining the quality of a scholarly journal. Employing a three-person independent judgment making panel, this study demonstrates the subjective nature of Beall’s Criteria by applying his Criteria to both OA and non-OA Library and Information Science journals (LIS), to demonstrate that traditional peer-reviewed journals could be considered predatory. Many of these LIS journals are considered as top-tier publications in the field and used when evaluating researcher’s publication history for promotion and tenure.