May

17

Transpersonal Assessments Practicum

May 17, 2004 | posted by Scott |

The Transpersonal Assessments Practicum (TAP) is a research project at the Institute of Tranpsersonal Psychology. I have been involved primarily in helping to create the online portion of the project… including designing the database, web-pages, and online assessments, as well as working on the implications of such a system in terms of tracking and ethical considerations.

The following is an early draft of the working paper “From Paper to Electrons”…

Introduction:

The Institute of Transpersonal Psychology (ITP) doctoral program emphasizes personal and professional development through an experiential learning model, which includes immersion in various whole-person, consciousness-raising experiential disciplines (e.g. Aikido, creative expression, meditation) in addition to the usual intellectual training in psychology. The Transpersonal Assessments Practicum (TAP) grew out of the Transformational Research Project (TRP), a multi-faceted, longitudinal research project designed to assess the experiences and changes of students going through the ITP doctoral program (Braud, Dufrechou, Raynolds, & Schroeter, 2000).

The study was designed to include data collection through a variety of means, including standardized instruments, surveys, qualitative self- and peer-reports, and a holistic-intuitive projective procedure (the Projective Differential). (Braud, et.al, 2000). Just as the ITP doctoral program was designed to provide opportunities for increased awareness and growth in the areas of bodily, emotional, spiritual, relationship and community, and creative expression training, the choice of instruments and other forms of data collection for the TRP was based on the intention to measure student experience and development in these same six areas, with an emphasis on transpersonal and psycho-social aspects. Therefore, a range of instruments and methods was required. (add into this discussion the four areas of emphasis that the Global group are working with, as an additional set of lenses for examining the data)

While the instruments being used in this research have undergone some changes over time, the study has continued to include several instruments and follow a particular protocol:

  1. standardized instruments administered three times (upon entrance to the program, and at the ends of the first and second years);
  2. holistic verbal and nonverbal connotative instruments administered three times (as above);
  3. two case studies - based upon peer interviews dealing with the students’ perceptions of changes in themselves (written at the end of the first and second years);
  4. two self-reports - based on self-reflection and dealing with students’ perceptions of changes in themselves (written at the end of the first and second years)

1995: Started with Res PHD; Fall 1997: added Res MA; Summer 2002 added 1st GPH cohort; Fall 2003 added Core Faculty & ITP Administrators

In 1997, the decision was made to expand the research project to include Masters level students; (At this time the project was also renamed, to the Transpersonal Assessments Practicum, and an already significant volume of data was set to grow. — add a statement about why William suggested the name change - basically to reflect what TAP had become, transcending to an experiential practicum); in 2003 core faculty, and administration were added. In addition to the volume of data itself, every participant had to be tracked, by both codenumber which could be cross-referenced only by select administrative staff, and by cohort (program and year).

Going online - the process:

With the prospect of vastly increased volumes of data, and the accompanying issues of participant tracking and follow-up, in 2002 the decision was made to move from a paper-based process of data collection to an online. (Prior to this, all tracking of participants and code numbers, verification of instrument completeness, and data entry was done manually.) Recognizing that this change faced issues of technology challenges, administrative adjustments, timing, and budget, it was decided to make the transition in phases.

Phase 1 - Spring (residential Post Assessments) & summer 2002 (inauguration of GPH assessments)

All of the standardized instruments, which previously had been created by simply photocopying masters, were converted into MS Word documents with form codings. This allowed all instruments to be distributed and collected via email. Students would receive the package including all standardized instruments via email, complete the instruments, save the document, and return it via email.

In addition, it was during this phase that the move was made from manual scoring of each individual instrument to the use of excel templates for batch scoring. While this phase allowed us to reduce the amount of paper moving back and forth between the administrators of the project and the participants, as well as the time and effort for manual scoring, it still required manual checking of every instrument submitted, data entry, and tracking and follow-up.

  • Formatting issues: Word & rtf vs WordPerfect, & Works
  • E-mail attachment issues: AOL, e-mail program capacity/size limitation issues

Phase 2 - fall 2002 - winter 2004 (Fall 2003 added Core Faculty)

All of the now MS Word based instruments were converted to html forms to be included on a website for online completion. A web-based scripting language (PHP) was chosen to parse the data input in the forms to create tab delimited text files of the data. The choice of PHP as the scripting language for dealing with the data input in the html forms was based largely on the fact that one of the research assistants for the project happened to have some familiarity with that language. In addition, he was familiar with a repository of pre-written scripts that were available through the open-source software community. The scripts used in this phase were, in fact, simply customized versions of scripts that are readily available and free for use, online.

Each instrument was treated as an entirely separate entity in this phase, and therefore there was a separate script and resulting text (data) file created for each one. An attempt to submit an incomplete form would result in an error message, and a request to go back and fill in the missing information. The tab delimited text files containing all data submitted, were downloaded by a project administrator to be imported into MS Excel for scoring. In order to achieve this with as little adjustment as possible, the Excel templates used for scoring had to be recreated to fit with the way the data was being collected. While data entry was no longer completely manual by this point, tracking of instrument completion remained a manual process requiring looking through the data to find what codenumbers were missing.

During this phase, the Informed Consent form was also made available online, using the same technologies. When a participant ’signed’ the informed consent (by selecting a button marked ‘Agree’ - participants were also given the opportunity to ‘Disagree’ and ask questions of clarification) their email address and name were submitted via email to the TAP administrators. Upon receipt of this email, the TAP administrator would send an email to the participant with information including the URL for the list of instruments to be completed, instructions for completion, and their codenumber (which would have to be included as a completed field with every instrument submitted).

Approx stats for 2002-2003:

  • Fall: 2 cohorts (PhD & MA)
    • Y1 PhD (@35): Basic pkg of 5 plus PDs & SDs (5)
    • Y1 MA (@30): Basic pkg of 5
  • Spring: 4 cohorts (PhD & MA)
    • Y1 & Y2 PhD (@ 65): Basic pkg of 5 + 3 plus PDs & SDs (5)
    • Y1 & Y2 MA (@ 50): Basic pkg of 5 + 3

E-mails

  • Assessment period introduction is sent via a mass e-mail to each cohort (4 basic e-mails)
  • Informed Consent generated an automated e-mail that then was responded to with detailed Assessments Set Instructions (About 115 incoming and outgoing e-mails)
  • Completed assessment (if 100% compliance)
    • PHD Pre generated 350 completion e-mails
    • MA Pre generated 150 completion e-mails
    • PHD Post generated 845 completion e-mails
    • MA Post generated 400 completion e-mails
    • Interview completions generated 115 e-mails

Follow up E-mails sent for:

  • Missing/late informed consent
  • Missing/late assessments
  • Missing/late interview completions
  • Missing/late end of year papers

For this phase of the project, TAP ‘borrowed’ web server space from the ITP Library. While this was satisfactory as a beginning strategy, it did mean that TAP did not have direct control over the web server, or direct access to technical support from the hosting company supplying the server. As a result, when there were problems with the server, ranging from issues of configuration that affected the functioning of the scripts, to complete downtime, the ability of the TAP project to collect data was compromised, and participants became frustrated.

Phase 3

As a result, in part, of an extended period of problems with the ITP Library web server, the decision was made for TAP to step out on its own in late fall 2003. A domain name was registered, a web hosting account and a security certificate were purchased. Since TAP was now to have its own webspace with greater control of the features and configuration of the server, and as the Phase 2 system would need some re-writing and testing to be sure it worked properly on the new server, it was decided to begin the process of moving to the next phase - a database driven system.

From the beginning of the process of taking the data collection online, the intention was to move to a system with a database back-end however, the timing of the move, and the time available for planning and ‘building’ of the third phase were partially dictated by the problems that had been experienced, and the need to have something stable in place prior to the next data collection period.

This phase of the online project required an aljost complete redesign. While much of the code that comprised the html forms that were the online versions of the instruments could be used, some changes and additions were required. The PHP scripts that were used to parse the data had to be rewritten completely. And a database had to be designed and built. While this represented a lot of work, it also represented significant opportunity.

The decision was made to use MySQL (an open-source relational database SQL - Structured Query Language - server) for the backend, and to continue with PHP as the scripting language - again, in part, due to the fact that a research assistant on staff had some experience with these and the open-source community in general (which could be used for technical assistance in addition to being a source for examples). (PHP - , which stands for “PHP: Hypertext Preprocessor” is a widely-used Open Source general-purpose scripting language that is especially suited for Web development and can be embedded into HTML. (http://www.php.net/manual/en/preface.php))

The nature of MySQL as a relational database, provided the opportunity to create a system that not only collected and stored the data collected through the submission of the online instruments, but to also track the completion of those instruments. As this phase progressed, it gradually grew to include an administration function, whereby TAP administrators could track and manage participant information, instrument completion, and import data directly from the database into Excel. (Rather than the prior e-mail confirmations)

With this iteration, all participant information (name, email address, code number, cohort, interviewer, and interviewee) were all added to the database prior to having participants access the site. Upon first access, participants would read and complete the Informed Consent, using the email address we have on file as an identifier. If the participant selects the ‘Agree’ button, they are taken to a page that provides them with their information, including their codenumber and password to access the online instruments.

By creating a system that requires participants to login, we also added the ability to track their access and completion of the assessments without requiring them to input their codenumber with each instrument, thus eliminating orphaned assessments. We were also able to add the ability for participants to submit their interview and self-reflection papers anonymously. With codenumber, cohort, and completion date automatically stored with all instrument data, we also now have the ability to track instrument completion by assessment, codenumber, or cohort.

In addition, by having everything stored in a database that can be accessed through a scripting language, we can more easily add additional functionality as the need arises.

Going online - the challenges:

The process of moving from a paper-based assessment process to an online process has not been without its challenges:

  • Budget considerations presented a challenge and lead to our initial decision to ‘borrow’ space on the ITP Library server, which served us well initially, but also created problems that were more difficult to resolve when technical problems with the server arose.
  • By chosing to go with ‘in house’ expertise, we were also limited by the technical abilities of that expertise, as that individual did not have any formal training in any of the technologies that have been used.
  • While the database system now in use is a significant advance over the initial online system, it may be limited in terms of future development by the design of the database, which is likely sub-optimal from a ‘leveraging the power of relational databases’ perspective.
  • Confidentiality and security considerations needed to be dealt with - ultimately a system requiring usernames and passwords, levels of administrative access, and encryption technologies served to provide confidence that the system is as secure as it can be.
  • While the database driven system is an advance in terms of storing and accessing of data, it was not designed with appropriate attention to the existing templates used for data scoring and analysis - as a result, unplanned and unanticipated adjustments have been required to get it all to work together.
  • Issues of browser compatibility (unfortunately, not all browsers conform to ‘accepted’ coding standards - thus some participants had difficulty accessing all functions in the early stages of each phase).
  • Issues of clarity of instructions, and changes in participant information (for example, if a participant had changed email addresses without informing ITP and then tried to complete the Informed Consent using a different email address than we had on file, they would be unable to proceed - this did lead to some confusion both on the participants’ part and ours as we tried to troubleshoot the problems).
  • Not everyone is equally comfortable with the internet - therefore, what some of us take for granted needs to be explained in more detail to those with less online experience. AOL users tended to need the jost support and were usually the individuals with minimal computer skills. They often found working with attachments to be problematic. WordPerfect users provided other compatibility issues related to skill level.
  • The first two phases of the project did not appropriately take into account all sides of the administration of the project - those of us working on getting the instruments and data collection online, were focused on getting the data collection and scoring to work, and did not pay sufficient attention to some of the implications the transition would have on the management of and communications with participants, and the tracking of instrument completion.

Going online - the benefits:

Despite the challenges and difficulties encountered along the way, the move to an online system of data collection has been a success overall. While it is not a completely automated system, it has served to simplify several aspects of the administration required for a large scale research project:

  • Tracking of instrument completion is no longer completely manual, and can be accomplished by logging into the administrative part of the site and clicking a button.
  • Instruments no longer need to be checked manually for readability, and full completion.
  • Manual scoring is no longer needed
  • Potential to handle increased volume of data without a corresponding increase in total RA hours required to manage the project

Going online - what we’ve learned:

While it is clear that moving to an online system of data collection has had significant benefits, the process of doing so has itself provided us with an opportunity to learn. Below is a list of things to think about when in the initial planning process for a similar project, some of which we did well, some of which we could have done better:

  • How will it affect the administrative side of the research project?
  • What technologies are available? And which will best suit the longer-term promise of going online?
  • ‘In-house’ vs. ‘expert’ developers contracted to do the design, building, and testing.
  • Purchase a pre-packaged system or develop a custom solution?
  • Data format (i.e. as data is collected, what format is it in, and how will it interface with the scoring/analysis systems you already have?)
  • Security issues - security of data (both stored, and in transmission), confidentiality
  • What will be the impact on participants? (Not everyone has the same level of experience or comfort with the internet - may require increased ‘technical support’ for participants)
  • Consider what additional technical skills will be required of, or training made available to RAs to manage the process once the ‘builders’ of the new system leave (particularly if you want to be able to upgrade the system or make changes to the instruments being used, etc. - also, the skill set for using programs such as Excel, and for accessing database tables is different than those required for manual checking and scoring of response sheets)
  • Plan, design, re-design, build, try to break it, redesign, rebuild, try to break it, fix it, etc., etc. (i.e. be prepared to spend significant time testing both in terms of whether it actually functions as expected, and in terms of useability)
  • While an online system may simplify many aspects of data collection and participant management and tracking in a large scale research project, do not expect it to eliminate the need for tracking and follow-up - and be prepared to add the need for technical support for some participants.

[Posted in Publications & Research & Technology]



Comments

Speak your mind...

Live preview:

CAPTCHA image

  • Blogs/Podcasts We Like

    • ITP Library Research Weblog - various research resources
    • Shrink Rap Radio - Podcasts by experienced therapist talking with notible folks in the field about everything you could think of.
    • The Merlin Show - A video podcast featuring Merlin Mann. Topics include: technology, communication, productivity, etc.


  • This site is:

  • And, this site is:

Close
E-mail It