Category: Conference Notes

Over the next several weeks, we will share notes from select NORDP 2020 virtual presentations. Check out the learning management system for details on all of the NORDP 2020 available presentations: https://nordp.mclms.net/en/package/list

1. Login with your NORDP member info.
2. Select the session you are interested in viewing.
3. Go to the Session Materials box and click on Materials which will take you to the presentation video and slides. 
4. The session will also appear in your personal course list for future viewing.

Presenters

• Justin Flory – The Biodesign Institute, Arizona State University
• Deborah Frank – Washington University School of Medicine
• Samarpita Sengupta – University of Texas Southwestern Medical Center
• Jessica Moon – Stanford University

Thanks to our session scribe, Daniel Campbell, Old Dominion University!

Key points from the session 

A RD scientist can leverage their own experiences by providing tailored education programs, personalized courses, and effective training for faculty investigators to communicate their research to the community at large.

Scientists in RD can help faculty develop a strong research question by deciding whether it is an important one to answer, if it feasible to answer, and to critically read the literature.

RD scientists can help raise potential reviewer criticisms, suggest alternative experiments, suggest controls, and evaluate data analysis plans.

In multidisciplinary proposals a RD scientist can serve as an extension of the PI that will allow them to focus their expertise. They also act as the common denominator to facilitate a multi-disciplinary team and help interconnect the science.

Scientists transitioning to RD face several challenges such as, imposter syndrome, understanding the institutional structure, and expanding the breadth of their editing beyond their core discipline.

What did you hear at this presentation that surprised you?

The imposter syndrome is a challenge faced by many folks who come from the postdoc world and who are now tasked with providing guidance to faculty who used to be their superiors in the higher ed hierarchy.

What was the most interesting question asked by an audience member?

Question – What motivates you and keeps you going in this field?

Answer – Seeing how many proposals are not funded and in many cases it is due to ineffective communication. Faculty have many great ideas & skills and we can help them with the communication element in our dual roles as scientists and skilled communicators.

Over the next several weeks, we will share notes from select NORDP 2020 virtual presentations. Check out the learning management system for details on all of the NORDP 2020 available presentations: https://nordp.mclms.net/en/package/list

1. Login with your NORDP member info.
2. Select the session you are interested in viewing.
3. Go to the Session Materials box and click on Materials which will take you to the presentation video and slides. 
4. The session will also appear in your personal course list for future viewing.

Presenters

• Jeff Agnoli, The Ohio State University
• Rebecca Bryant, OCLC Research
• Nina Exner, Virginia Commonwealth University

Thanks to our session scribe, Daniel Campbell, Old Dominion University!

Key points from the session 

Libraries are not ends in themselves, they are shifting from collection centric model to engagement-oriented model supporting teaching, learning and research workflows

Libraries are becoming partners in the research enterprise. They enhance researcher productivity, facilitate analysis of research, and make research visible to the scholarly community and beyond.

Librarians are very skilled at finding the literature; awareness of trending topics within fields, crossing disciplines for potential impact on other areas, alerting services to keep up with new literature, and citation management.

The Ohio State University has developed a University Libraries Research Commons which serves as a neutral physical/virtual space; offers consultations, education & training, referrals, and space to showcase.

Libraries are often viewed as neutral players and can be very helpful with communication across a team of researchers.

What did you hear at this presentation that surprised you?

Librarians can use bibliometrics to help PI’s strategically build the impact of their publications over time.

What was the most interesting question asked by an audience member?

What type of librarian would help with competitive intelligence?

Response: Their title varies depending on the institution, but examples include Research Impact Librarian, Bibliometrics/impact, Business subject specialist, Scholarly communications, or Metrics Librarian.

What else from this session should NORDP members know?

The presenters used a virtual whiteboard during the Q&A portion and it was a great example of live interaction and sharing of ideas on RD partnerships.

Over the next several weeks, we will share notes from select NORDP 2020 virtual presentations. Check out the learning management system for details on all of the NORDP 2020 available presentations: https://nordp.mclms.net/en/package/list

1. Login with your NORDP member info.
2. Select the session you are interested in viewing.
3. Go to the Session Materials box and click on Materials which will take you to the presentation video and slides. 
4. The session will also appear in your personal course list for future viewing.

MC: Jan Abramson, NORDP Fellow

Thanks to our session scribe, C. Scott Balderson, University of Utah!

Lightning Talks and Presenters:

The Mentoring Committee’s resident poet, Eric Dickey, explains how he uses poetry to create a shared vocabulary and experience with collaborators, as well as utilizing it as metaphor or simile to help explain complex concepts and assist scientific researchers, who may not be avid writers, find greater appreciation for and ease with the creative process necessary to write proposals or project reports. His talk provides some great examples that he uses regularly, and he urges participants to try similar narrative forms like songs and movies that can help audiences open discussion.

2. Mentoring as a Professional Development Journey, Angela Jordan – University of South Alabama, ajordan@southalabama.edu

When Angela Jordan first contemplated becoming a mentor for NORDP, she thought that she had few of the qualities she associated with the role in the context of this organization: she’d been in RD less than 5 years; her background was humanities, not science; she “only” had a Master’s degree; and she’d only been a mentor in our program for a year. Still, she decided to make the leap, and in doing so, she discovered a wealth of things. She had much to offer (her framing of elements of the experience in terms of “appreciative inquiry” and having a “growth mindset” are helpful constructs) yet realized that becoming a mentor was not an end point but a continuing part of her own development journey.

3. Following the Leaders, Erica Pitre – University of Louisiana at Lafayette, erica.pitre@louisiana.edu

Erica Pitre came to academic Research Development after years many years doing equivalent work in industry and building her skills and approaches to suit the needs of that field. This presentation is a brief examination of her first year in making the transition to academic RD, the obstacles she faced using the tools and techniques she’d refined in her previous career, and the wisdom to cope and prevail that she found in NORDP and its mentoring program.

4. Piloting Accountability Groups for Peer Mentoring Among Early Career Faculty, Kathy Partlow – University of Nebraska-Lincoln, kcpartlow@unl.edu

Ever the lover of data and the power that inferences from them can give to aid in mentoring program design and execution, Kathy Partlow unveils a pilot program for peer mentoring faculty around grant writing at her institution, complete with preliminary results and her thoughts on the meaning behind some of those as well as tips on ways other programs could be structured differently for different results. You could describe this as a natural experiment, in that Kathy allowed faculty groups different options with regard to frequency of meetings and type of accountability and then got to see what, if any, impact this had on faculty proposal submissions. Interesting results!

5. A Concrete Example of the Benefits of Being a Mentor: Creating a CV for Research Development, Jennifer Glass – Eastern Michigan University, jglass5@emich.edu

A common concern for some looking to enter into the Mentoring Program as a mentee is focus: how will you use the experience with the mentor? Do you want to refine your skills? Increase your network? Develop leadership? Explore new areas of RD? Transition to a new role? All of the above? It can be overwhelming and paralyzing. Jennifer Glass uses the example of working with a mentee on the very specific shared goal of documenting their RD activities and updating their CVs with a real focus on Research Development. Amazingly, in this very short presentation, she takes us through the process in a way that makes the (perhaps daunting) task feel completely replicable. I’m watching this one a few more times, until my own resume is done!

6. From Novice to Expert in RD – Mentor/Mentee Perspectives, Paula Carney – Loyola University Chicago, pcarney2@luc.edu

As part of establishing good mentoring relationships, it can be helpful to know what “level” the individuals are operating at in terms of research development. Paula Carney puts her background in psychology (with nods to the developmental psych giants) to good use by adapting a paradigm for RD levels, complete with practical examples, that allows users to ascertain where they are and, more importantly for the mentoring context, the type of professional activities and achievements they might see in mentors having already attained the level they would like to achieve. I think this topic is especially valuable as we consider whether formalized Certifications in our field would be beneficial.

7. Mentoring: Viewing the Engagement from the Mentee’s Perspective, John Barfield – Tennessee State University, jbarfield@tnstate.edu

If the previous talk was heavily weighted to an RD application of psychology, John Barfield’s topic takes us down a more philosophical path. Throughout his years, John has accumulated a view of the mentoring relationship as being mentee-focused, yet acknowledging that it is very much a two-person relationship that takes continual effort. As with most of us, he mentors in a variety of ways and settings to different types of mentees, and he has used this diversity of experience to distill some principles of being a good mentor (by always trying to see through the eyes of the mentee) and presents this philosophy in colorful and effective similes and examples.

8. Mentoring with Vulnerability, Hilda McMackin, Vanderbilt University, hilda.mcmackin@vanderbilt.edu

Hilda McMackin builds on the previous two talks, blending a philosophy of mentoring with some approaches like active and empathic listening that were developed in counseling settings. Her inspiration came from the book Dare to Lead by Renee Brown, who coins the concept of “rumbling with vulnerability.” Hilda applies that idea for mentoring as a way of establishing trust and connection necessary to the deepest relationships. She advocates having the courage to share, to connect what may not be similar experiences but different experiences that elicit the same emotions. Show the same interest and attention you want another person to have for you and your ides. Be vulnerable and hang in there through things that may not always be comfortable. In that way, trust is developed that allows true curiosity and real expectation setting to flourish without a façade of limits imposed by the fear of sharing ourselves with the person we’re mentoring or being mentored by.

Thanks for your interest in the Mentoring Lightning Storm. Feel free to reach out to any of the presenters, or email mentorprogram@nordp.org for more information.

Over the next several weeks, we will share notes from select NORDP 2020 virtual presentations. Check out the learning management system for details on all of the NORDP 2020 available presentations: https://nordp.mclms.net/en/package/list

1. Login with your NORDP member info.
2. Select the session you are interested in viewing.
3. Go to the Session Materials box and click on Materials which will take you to the presentation video and slides. 
4. The session will also appear in your personal course list for future viewing.

Presenters

• Sandra Holden, Ph.D. – Assistant Director, Stanford Research Development Office
• Babette Heyer, Ph.D. – Director, Research Strategy Development at the Stanford Cancer Institute
• Sarah Ott – Senior Grants Consultant, Hanover Research

Thanks to our session scribe, Daniel Campbell, Old Dominion University!

Key points from the session 

Institutional level incentives can involve seed grants, faculty release time, recognition in the promotion & tenure process, priority access to RD staff and Red Team review. RD level often does not have the authority to set these items up, but we can support implementation of these programs by administering seed grants, coordinating Red Team reviews and prioritizing support for team science proposals.

RD level strategies can involve encouraging faculty to meet others from different fields, share bios & research interests on internal databases, and RD staff can foster connections among PI’s you work with as an individual.

Important first steps when considering a project include: a detailed review of FOA; ask does the PI have the time to do it right; confirm institutional support; solicit funder input; and establish partnerships.

It is important for the team to be thinking not only about how to write the proposal, but how they will work well together. Team Science Guiding Questions to consider include:

• What is your rationale?
• Are you ready to collaborate?
• How will you address and manage essential team processes?
• Do you have the technology and resources required?
• How will you communicate and coordinate?
• How will team leadership, management, and administration look?
• How will you resolve conflict?
• How will you evaluate your collaboration?

It is important to have an institutional level advocate ideally who can be a supporter of the project when hurdles or issues develop.

What did you hear at this presentation that surprised you?

When developing the project timeline use the last 1/3 to review combined elements.

What was the most interesting question asked by an audience member?

Is there an optimal number of team members to make a proposal competitive? 

Response: There is no ideal number. It is really determined with the scope of and what is required by the individual project. The key is to have engaged participants, engagement not number is the most important aspect.

What else from this session should NORDP members know?

There are a lot of great resources discussed by the presenters including project management tools, proposal writing resources, document management considerations, and networking activities.

There’s also a Part 2 that is an informal Q&A session that is a follow-up to Part 1.  

Slides: They’re doing WHAT!?! Intelligence Gathering in Higher Ed

Presenters:

• Karen Walker, Arizona State University
• Jamie Welch, Arizona State University
• Linda Galloway, Elsevier

Thanks to our session scribe!

Five key points from the session:

• ASU’s RD office includes a full-time person in charge of “competitive intelligence.“
• ASU defines competitive intelligence as the “ethical collection and analysis of information to anticipate competitive activity, see past market disruptions and dispassionately interpret events.”
• ASU RD staff works with Elsevier’s Scival to identify areas of strength, then continues to study these strengths and prepares documents to help faculty maximize these strengths and build collaborations.
• SciVal allows researchers and administrators to visualize research performance, benchmark relative to peer institutions, develop strategic partnerships, identify and analyze new, emerging research trends, and create uniquely tailored reports.
• ASU’s RD tagline is “Knowledge Enterprise Development.”

What did you hear at this presentation that surprised you?

ASU’s investment in publications that provide details on priority areas for the university.

What resources did you discover at this presentation?

• Altmetric: www.altmetric.com
• SciVal: https://www.elsevier.com/solutions/scival

What else from this session should NORDP members know?

Successful institutions take time and invest to support faculty and build capacity.

Slides: Facilitating Innovative Research 

Presenters:

• Donnalyn Roxey, Knowinnovation
• Andy Burnett, Knowinnovation

Thanks to our session scribe, Christina Howard, Texas A&M University!

As research development professionals, how can we make meetings more interesting, productive, and effective? How do we promote creative thinking? At the Effective Meeting Facilitation workshop, we entered the room to find an array assortment of squishy ducks, Slinkys, and other fidget toys on chairs throughout the room. The presenters used these along with a variety of engaging means to generate ideas and discussion among audience members. They emphasized: When designing a meeting it is important to consider what type of environment might facilitate creativity, and to recognize that different creative preferences can still achieve similar outputs.

Effective Meeting Facilitation, Part One

The focus was on the various dimensions of a creative climate and on specific ways RD professionals can encourage and facilitate each of these. Dimensions include: dynamism and liveliness; freedom; risk taking; playfulness and humor; idea time; idea support; trust and openness; conflict; debate; and challenge.

Tips for the RD professional

Some practical ways to encourage dynamism and liveliness are to move people around: get them talking to new people or interacting in new ways. For example, hold a meeting in a new or different location. One way to incorporate playfulness and humor is to allow for “mind wandering” through the use of fidget toys.

Assist with providing idea time and idea support by leveling the playing field: making sure everyone’s ideas are heard, deferring judgment (both positive and negative), and separating out divergence from convergence by listening to all ideas before reviewing and matching them with goals and objectives. Additionally, allowing time for ideas to incubate (i.e., thinking and regrouping) is critical rather than trying to cram everything all into one meeting.

Some suggestions to make meeting environments more creative were using fidget toys, holding meetings off-campus, using sticky notes, process mapping, adding a little bit of “interest” to the meeting space (i.e. in the form of potted plants).

Effective Meeting Facilitation, Part Two

The focus was on strategies for conducting virtual meetings effectively. Virtual collaboration can decrease our carbon footprint and increase affordability… if it works. There are multiple ways to collaborate virtually across space (i.e. Zoom), time (i.e. bulletin boards, computer screens for asking and answering questions), or a combination of both (i.e. Google Docs).

Tips for the RD professional

Ideas labs use a combination of virtual and face-to-face meetings to engage individuals. Have everyone participate virtually in an initial “ideas lab.” This can help gauge who will engage – and who should be present at the table in a face-to-face meeting.

“Randomized coffee talks” (RCTs). These have the goal of getting people together who might otherwise never meet or interact, and provide them with a small incentive for doing so (i.e. coffee). For example, RCTs could be a good pre-meeting exercise to help meeting participants engage with each other prior to a larger proposal meeting.

An important factor to consider when organizing a virtual meeting is the length of time, in part because the cognitive load is larger in a virtual space than in a physical space. One way to alleviate this would be to schedule shorter meeting times with longer breaks in between when planning a virtual meeting. Also, encouraging meeting participants to go to an alternate location for the virtual meeting (rather than just sitting in their offices) may help participants engage more fully rather than treating the virtual meeting as if they were watching a virtual webinar.

Regardless of the meeting format (in-person or virtual), however, accessibility is key. The presenters emphasized that either all-physical or all-virtual meetings tend to work more effectively than a mix of the two (part virtual and part in-person).

Together, these two workshops provided valuable ideas and resources for helping meeting participants fully engage, whether the meeting is held in-person or virtually. KnowInnovation (fir.hub.ki) has a number of resources that can help with making meeting environments more creative, including tools, blogs, drop-ins, a virtual idea board, and an email short course on creativity and skill development, as well as another email short course on creative thinking and problem-solving.

Slides: Employing Tactical and Strategic Approaches to Help Faculty Maximize Broader Impacts

Presenters:

• Danielle Mazzeo, American Museum of Natural History
• Nathan Meier, University of Alabama at Birmingham
• Tisha Mullen, University of Nebraska-Lincoln
• Stephanie Hensel, University of Michigan School of Education

Thanks to our session scribe, Paige Belisle, Harvard University!

This presentation explored how different institutions address Broader Impacts requirements during the proposal development process. The National Science Foundation defines Broader Impacts as activities associated with sponsored research that will benefit our society and world. Other Federal agencies require similar components that will have impact outside of traditional academia. Meeting Broader Impacts requirements can be accomplished through the research/project itself; through activities directly related to the research/project; or through activities complementary to the research/project.

Case Study: American Museum of Natural History, New York City

This portion of the presentation focused on how the museum uses intra-institutional connections to maximize Broader Impacts activities. The museum aims to create & support interdisciplinary partnerships; leverage other departmental efforts to support new Broader Impact activities; and develop Broader Impact activities that are replicable and scalable. This is carried out by establishing links between its library and archives, cyberinfrastructure, graduate students, research projects, youth programs, and exhibitions, all of which also serve the museum’s central mission.

Case Study: University of Michigan School of Education

This section of the presentation discussed a university center which addresses Broader Impacts. The University of Michigan School of Education’s Center for Education Design, Evaluation, and Research (CEDER) “advances equity and excellence in education by providing access to high quality design, evaluation, and research services through collaborations with university, school, and community partners.” CEDER helps faculty during the proposal development, project implementation, and research dissemination stages by offering design, evaluation, and research development services. Through consultations, CEDER helps faculty consider how to approach Broader Impacts activities in K-12 classrooms.

Case Study: University of Nebraska-Lincoln

This portion of the presentation explored how the University of Nebraska-Lincoln uses a matrix approach for Broader Impacts support by providing something for nearly everyone in the community: Pre-K to seniors, formal and informal activities, and local to global impact. Their group emphasizes that there are no cookie-cutter approaches that guarantee Broader Impacts success. The presentation discussed the importance of leveraging existing partnerships and infrastructure. It  argued that a faculty member doesn’t need to reinvent themselves to successfully implement Broader Impacts activities; rather, they should focus on how their existing connections, passions, and projects might be expanded to meet the requirements.

What did you hear at this presentation that surprised you?

I did not realize that the American Museum of Natural History in New York functions as a PhD-granting research institution, and it was very interesting to hear their unique perspective on Broader Impacts.

What resources did you discover at this presentation? 

E-resources from the presentation can be found here: http://bit.ly/2ZvBtqz

The presenters mentioned that the National Alliance for Broader Impacts (NABI) offers many additional resources: https://broaderimpacts.net

What were the most interesting questions asked by an audience member, and what was the presenters’ responses?

Question: What if PIs want to focus only on training graduate students for their Broader Impacts?

Answer: While this is a very important aspect, it is typically not adequate to only focus on training graduate students in the Broader Impacts section. Generally, the proposal will need to have other Broader Impacts activities as well.

Question: How do you encourage faculty to visit a curriculum development office (or other relevant Broader Impacts resource) early in the proposal development process?

Answer: A helpful website can be a good first step in advertising your services. You can also create a “roadshow” presentation to bring to existing meetings, especially for schools and departments that may need extra help.

Question: What are the most common critiques regarding Broader Impacts in unsuccessful proposals?

Answer: A comment the presenters have frequently seen is that the Broader Impacts were interpreted by the reviewers as too broad or ambitious in scope. Proposal writers should focus on the quality of their Broader Impacts activities rather than the quantity of activities they are proposing. Goals should be concrete and attainable, not aspirational. Broader Impacts partnerships should be authentic.

Slides: Building the RD Professional’s Toolbox and Skills for Developing Project Evaluation Plans for Grant Proposals

Presenters:

• Katie Allen, Kansas State University, Office of Educational Innovation and Evaluation
• Morgan Wills, Kansas State University, Office of Educational Innovation and Evaluation
• Makenzie Ruder, Kansas State University, Office of Educational Innovation and Evaluation

Thanks to our session scribe, Paige Belisle, Harvard University!

This highly engaging NORDP session presented by Kansas State University’s Office of Educational Innovation and Evaluation team offered attendees tools and resources for considering grant proposal evaluation plans. The session explored terms and ideas common to evaluation plans and provided guidance for Research Development professionals who are new to this aspect of proposal development. Defining evaluation as determining the value or worth of a project, the presenters suggested that evaluation can also be described as a means to document the success of a particular program.

The presenters explained that first, you will need to determine criteria indicators for making your assessment. Many funding agencies will have a list of required documents and specific instructions for what is required within the evaluation plan. An internal evaluation will require the research team to evaluate their own work, while an external evaluation will be conducted by someone independent from the project. Essentially, an evaluation asks whether a project’s overall purpose is being met, given the requirements of the program solicitation. For example: do the PIs need to produce publications, or gather and report on a specific form of data? Have they met these goals successfully?

As a Research Development professional, there are many ways that you can help the PI prepare their evaluation plan. You can help a PI identify the supplemental documents that they will need to gather to be compliant with the solicitation. The RFP language used to describe similar types of evaluation plan components can vary from sponsor to sponsor, as will the agency’s specific guidance for things to include within those components. This sponsor-specific language and terminology will be important to ensuring the proposal’s competitiveness. PIs may need assistance in considering what to budget for their project’s evaluation. The presenters advised that an external evaluation budget should be roughly 10% of the total proposed budget for the project. In the budget justification, you might suggest that the PI specify why the evaluation will cost what they are proposing. For example, will the external evaluator need to travel to the PI’s lab to assess the project? You can also provide assistance by reviewing the proposal’s project communication plan and timeline.

What did you hear at this presentation that surprised you?

Because I do not personally assist faculty with budget preparation, I was interested to learn that an external evaluation budget is often roughly 10% of the total proposal budget.

What resources did you discover at this presentation? 

The presenters put together a comprehensive list of links and additional resources on the topic of evaluation, which can be found here: http://bit.ly/OEIE_NORDP

What was the most interesting question asked by an audience member, and what was the presenter(s)’ response?

One audience member asked how to approach the topic of developing an evaluation plan with a faculty member who may be apprehensive about the process of an external evaluation. The presenters explained that one way to frame evaluation is to think of evaluators as “critical friends” who can help bring a project to the next level. The presenters explained that, in their own experience, when a PI works with an evaluator once, they begin to appreciate the process and benefits of evaluation.

Presenters:

• Jessica Venable, McAllister & Quinn
• M.S. (Peg) AtKisson, AtKisson Training Group, LLC
• Joanna Downer, Duke University School of Medicine
• Michael Gallo, University of California-Irvine

Thanks to our session scribe, Samarpita Sengupta, PhD, UT Southwestern Medical Center!

Have you ever been in a situation where a faculty member wants to submit an application for a big grant that is due in a month? Who am I kidding, you are research development professionals, of course you have!

How do you make a go-no-go decision? What are your criteria? This was the topic of the talk given by the four presenters listed above at the 11^th Annual NORDP conference in Providence. Each speaker approached the topic from their individual research development perspectives.

Large, complex grants, as defined by the presenters, could be “Super-big institutional opportunities;” large multi-investigator, multi-site, multi-disciplinary projects, some involving construction, renovation or building centers; or they could have non-standard requirements.

There are several points to think about while making go-no go decisions on such large grants, especially in a time sensitive situation, such as the presence of internal resources, whether the project met unmet needs, whether external resources could be tapped into, what is the return on investment, and whether the benefits outweigh the costs. Internal resources should be factored in when making such a decision, such as project management, team facilitation, graphic design and domain-specific expertise, proposal building and administrative capabilities. In cases where the internal resources are tapped out or otherwise unavailable, external resources, such as consulting firms or freelance RD personnel, can be brought in to fill those unmet needs. A hybrid model, whereby the internal team works with an external team to submit the application, are particularly useful when the application is a priority of the institution and a no-go decision is impossible to make.

Every time an external consultant/freelancer is brought in, it is important to weigh the costs with the benefits, especially at smaller institutions. When bringing in external consultants, a shared vision for success is necessary. It is important to set expectations early, asking for references for each consultant to answer questions like: have they worked with similar projects before, are they reliable and timely, do they set and manage expectations, do they have good communication skills across diversity in teams, cultures and discipline, are they team-players, do they make you look good, and do they have a good network that you can tap into.

If tapping into external resources is not a possibility, then the internal team needs to reevaluate the go-no-go decision tree. At times, it is useful to bring in an external team just to get an outsider perspective and to reiterate the no-go decision.

In conclusion, the presenters reiterated that having clear SOW with external consultants, starting early with internal team with planning, idea generation, brain storming, and being cognizant of your own limitations can help with these decisions. The ability to create a dedicated team of internal and/or external contributors and seek out highly specified individuals to fill gaps in expertise is key to successful go-no-go decision making.